Papers with annotation schemes
A Joint Named-Entity Recognizer for Heterogeneous Tag-sets Using a Tag Hierarchy (P19-1)
Copied to clipboard
| Challenge: | Named-entity recognition (NER) has seen significant progress with the application of Neural Networks to the task. |
| Approach: | They propose to use a given tag hierarchy to jointly learn a neural network that shares its tagging layer among all tag-sets. |
| Outcome: | The proposed model outperforms models that combine independent models and multitasking approaches in a domain adaptation for named-entity recognition task. |
Active Learning Design Choices for NER with Transformers (2024.lrec-main)
Copied to clipboard
Robert Vacareanu, Enrique Noriega-Atala, Gus Hahn-Powell, Marco A. Valenzuela-Escarcega, Mihai Surdeanu
| Challenge: | In the field of natural language processing, active learning is a technique that is used to decide which examples are worth annotating . a number of studies have focused on sequence classification, text classification, question answering, and question answering. |
| Approach: | They propose two different approaches to deal with partially-annotated sentences . they propose an annotation scheme that can be used to train with tokens . |
| Outcome: | The proposed approaches achieve comparable or better performance than sentence-level annotations with a smaller number of annotated tokens. |
Towards Agile Text Classifiers for Everyone (2023.findings-emnlp)
Copied to clipboard
Maximilian Mozes, Jessica Hoffmann, Katrin Tomanek, Muhamed Kouate, Nithum Thain, Ann Yuan, Tolga Bolukbasi, Lucas Dixon
| Challenge: | Text-based safety classifiers are widely used for content moderation and increasingly to tune generative language model behavior. |
| Approach: | They propose to use small, targeted datasets to train safety classifiers using small, iterative datasets that can be quickly developed for a particular policy. |
| Outcome: | The proposed method can be quickly developed for a specific policy with a labeled dataset of as few as 80 examples. |
SACR: A Drag-and-Drop Based Tool for Coreference Annotation (L18-1)
Copied to clipboard
| Challenge: | Several annotation strategies have been proposed to balance scientific needs with annotation speed. |
| Approach: | They introduce SACR, an easy-to-use coreference chain annotation tool . it is used to annotate large corpora for natural language processing applications . paper compares several annotation schemes implemented in existing tools . |
| Outcome: | The proposed tool is used to annotate large corpora for natural language processing applications. |
MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction (2022.emnlp-main)
Copied to clipboard
Xiaozhi Wang, Yulin Chen, Ning Ding, Hao Peng, Zimu Wang, Yankai Lin, Xu Han, Lei Hou, Juanzi Li, Zhiyuan Liu, Peng Li, Jie Zhou
| Challenge: | Existing datasets only cover limited relation types at once, which prevents models from taking full advantage of relation interactions. |
| Approach: | They construct a large-scale human-annotated ERE dataset with improved annotation schemes to address these drawbacks. |
| Outcome: | The proposed dataset is larger than existing datasets of all the ERE tasks by at least an order of magnitude. |
The Metalogue Debate Trainee Corpus: Data Collection and Annotations (L18-1)
Copied to clipboard
Volha Petukhova, Andrei Malchanau, Youssef Oualil, Dietrich Klakow, Saturnino Luz, Fasih Haider, Nick Campbell, Dimitris Koryzis, Dimitris Spiliotopoulos, Pierre Albert, Nicklas Linz, Jan Alexandersson
| Challenge: | Argumentation is an important component of human intelligence and is used to train lawyers and citizens in legal domains. |
| Approach: | They describe the Metalogue Debate Trainee Corpus (DTC) which contains data on motion and speech capture devices and semantic annotations. |
| Outcome: | The metalogue Debate Trainee Corpus (DTC) was developed to facilitate the design of instructional and interactive models for the Virtual Debate Coach application. |
Towards Continuous Dialogue Corpus Creation: writing to corpus and generating from it (L18-1)
Copied to clipboard
| Challenge: | Existing methods to create dialogue corpora annotated with interoperable semantic information are based on ISO standard data models and tools. |
| Approach: | They propose to use a corpus as a shared repository for analysis and modelling of interactive dialogue behaviour and for implementation, integration and evaluation of dialogue system components. |
| Outcome: | The proposed method is applied to the design of two multimodal interactive applications - the Virtual Negotiation Coach and the Virtual Debate Coach. |
From Form to Meaning: The Case of Particles within the Prague Dependency Treebank Annotation Scheme (2025.coling-main)
Copied to clipboard
| Challenge: | Discussions on an appropriate annotation scheme for large and complex information are ongoing . multi-layer system allows a comprehensive description of relations between morphological properties, syntactic function and expressed meaning. |
| Approach: | They propose a multi-layer annotation scheme for the Prague Dependency Treebank . they propose morphological properties, syntactic function and expressed meaning as multi-layered systems . |
| Outcome: | The proposed scheme is sound and serves well for complex annotations. |
Dependency Parsing for Spoken Dialog Systems (D19-1)
Copied to clipboard
| Challenge: | Dependency parsing of conversational input can help to understand dialogs . currently available annotation schemes do not adapt well to spoken human-machine dialogs. |
| Approach: | They propose an annotation scheme that extends Universal Dependencies guidelines to spoken dialogs. |
| Outcome: | The proposed scheme disambiguates relationships between entities extracted from dialogs . it is better than existing models on public datasets and fine-tuned on ConvBank data . |
Partial Or Complete, That’s The Question (N19-1)
Copied to clipboard
| Challenge: | Existing annotation schemes aim at acquiring completely annotated structures, but partial annotations can be costly and hinder learning. |
| Approach: | They propose a method to find out that learning from partial structures can sometimes outperform learning from complete ones. |
| Outcome: | The proposed method outperforms existing methods in three different structured learning tasks. |
Comparing learnability of two dependency schemes: ‘semantic’ (UD) and ‘syntactic’ (SUD) (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Several studies have suggested that choosing syntactic criteria for assigning heads in dependency trees improves the performance of dependency parsers. |
| Approach: | They propose to use syntactic criteria to assign heads to dependency trees to improve the performance of dependency parsers by using a selection of 21 treebanks. |
| Outcome: | The proposed approach favours content words over function words as heads of dependency relations, while the other favours syntactic heads. |
Rethinking the Agreement in Human Evaluation Tasks (C18-1)
Copied to clipboard
| Challenge: | In natural language processing, IAA is often viewed as a means of assessing the quality of data on a task, in particular, the reliability. |
| Approach: | They propose a new approach to use agreement metrics in natural language generation evaluation tasks to reduce subjective bias. |
| Outcome: | The proposed approach is based on the inter-annotator agreement (IAA) of natural language generation tasks. |
ISO-Standard Domain-Independent Dialogue Act Tagging for Conversational Agents (C18-1)
Copied to clipboard
| Challenge: | Existing methods for DA annotation are incompatible with each other and do not cover all aspects necessary for open-domain human-machine interaction. |
| Approach: | They propose to map publicly available corpora to a subset of the ISO standard and create a task-independent training corpus for DA classification. |
| Outcome: | The proposed method can train a domain-independent DA tagger on out-of-domain conversational data and achieve robustness across different DA categories. |
Annotations Matter: Leveraging Multi-task Learning to Parse UD and SUD (2021.findings-acl)
Copied to clipboard
| Challenge: | Multi-task learning (MTL) has shown promise in various NLP tasks such as semantic dependency parsing. |
| Approach: | They propose to use two parallel treebanks to improve parsing performance. |
| Outcome: | The proposed model is based on two parallel treebanks with similar annotation schemes but differing in linguistic annotation preferences. |
Joint Turn and Dialogue level User Satisfaction Estimation on Multi-Domain Conversations (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to estimate turn and dialogue level user satisfaction employ hand-crafted features and rely on complex annotation schemes, which reduce generalizability of the trained models. |
| Approach: | They propose to use an adaptive multi-task loss function to minimize hand-crafted features to estimate user satisfaction at turn level from an end user perspective. |
| Outcome: | The proposed model improves on 28 Alexa domains, two dialogue systems and three user groups on a set of user-generated dialogues from 28 Alexia domain and 28 Alexis domains. |
Cross-type French Multiword Expression Identification with Pre-trained Masked Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Multiword expressions (MWEs) have linguistic features that distinguish them from regular word groupings. |
| Approach: | They propose a combination of two systems that learn verbal multiword expressions and non-verbal MWEs to improve performance on a cross-type dataset . |
| Outcome: | The proposed system improves the F1 score on a french treebank with VMWEs and nVMWES training data. |
Does Recommend-Revise Produce Reliable Annotations? An Analysis on Missing Instances in DocRED (2022.acl-long)
Copied to clipboard
| Challenge: | Document-level relation extraction is a challenging task as it requires reasoning across multiple sentences. |
| Approach: | They propose to use a recommend-revise scheme to reduce the workload of annotators by providing them with candidate relation instances from distant supervision to supplement and remove relational facts. |
| Outcome: | The proposed dataset is the first large-scale and human-annotated dataset for relation extraction. |
EMAD: A Bridge Tagset for Unifying Arabic POS Annotations (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing tagsets for Arabic are difficult to combine due to the diversity of their features. |
| Approach: | They propose an Arabic Extended Morphological Analysis and Disambiguation Tagset which facilitates conversion and unification of Arabic tagsets. |
| Outcome: | The proposed tagset facilitates conversion and unification of different tagsetes used to annotate Arabic corpora. |
Towards Evaluation of Cross-document Coreference Resolution Models Using Datasets with Diverse Annotation Schemes (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing cross-document coreference resolution (CDCR) datasets contain event-centric coreference chains of events and entities with identity relations. |
| Approach: | They propose to use a phrasing diversity metric to evaluate lexical diversity of CDCR datasets . they propose to combine CDCR annotation schemes with multiple properties of the coreference chains . |
| Outcome: | The proposed phrasing diversity metric evaluates the CDCR datasets with higher precision. |
Annotating Zero Anaphora for Question Answering (L18-1)
Copied to clipboard
| Challenge: | a large dataset of zero pronouns has been constructed to identify adjunct zero anaphoras . a lack of a dataset covering them has limited our ability to annotate them exhaustively . |
| Approach: | They propose to annotate adjuncts marked by -de in Japanese and a second scheme to annnotate them in a more direct manner. |
| Outcome: | The proposed annotation schemes are more accurate than the first one. |
Automatic Argument Quality Assessment - New Datasets and Methods (D19-1)
Copied to clipboard
Assaf Toledo, Shai Gretz, Edo Cohen-Karlik, Roni Friedman, Elad Venezian, Dan Lahav, Michal Jacovi, Ranit Aharonov, Noam Slonim
| Challenge: | 6.3k arguments were collected from contributors of various levels, and are released as part of this work. |
| Approach: | They propose to use a language model to annotate arguments for argument ranking and argument-pair classification. |
| Outcome: | The proposed methods outperform state-of-the-art methods in the argument ranking task and argument-pair classification task. |
Annotating Mentions Alone Enables Efficient Domain Adaptation for Coreference Resolution (2023.acl-long)
Copied to clipboard
| Challenge: | Recent results show that annotating mentions is twice as fast as annotation of full coreference chains. |
| Approach: | They propose a method for efficiently adapting coreference models using only mentions in the target domain without increasing annotator time. |
| Outcome: | The proposed method improves average F1 without increasing annotator time. |
Annotating for Hate Speech: The MaNeCo Corpus and Some Input from Critical Discourse Analysis (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing methods for detecting hate speech are based on the problem of identification, but there is no clear definition of hate speech. |
| Approach: | They propose a multi-layer annotation scheme for the detection of hate speech in a web 2.0 corpus . they propose to use a binary hate speech classification to identify hate speech . |
| Outcome: | The proposed scheme is piloted against a binary hate speech classification and appears to yield higher inter-annotator agreement. |
GerCCT: An Annotated Corpus for Mining Arguments in German Tweets on Climate Change (2022.lrec-1)
Copied to clipboard
| Challenge: | Recent work on annotated resources focused on single argument components, i.e., claim or evidence. |
| Approach: | They propose to annotate a German climate change argument corpus using sarcasm and toxic language to facilitate filtering out non-argumentative content. |
| Outcome: | The proposed corpus is the first to be annotated for argumentation, sarcasm and toxic language. |
TIARA: A Tool for Annotating Discourse Relations and Sentence Reordering (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing tools for discourse relations and sentence reordering are difficult to use and clutter the display. |
| Approach: | They propose to use TIARA to simplify the annotation process by offering interactive visualisation, including coloured links, indentation, and dual-view. |
| Outcome: | The proposed tool simplifies the annotation process and offers visualisations including coloured links, indentation, and dual-view. |
Negation Scope Conversion: Towards a Unified Negation-Annotated Dataset (2024.lrec-main)
Copied to clipboard
| Challenge: | Negation scope resolution models that use pre-trained language models perform worse when fine-tuned on a combined dataset. |
| Approach: | They propose to automatically convert the negation scopes of BioScope and SFU to those of Sherlock and merge them into a unified dataset. |
| Outcome: | The proposed method improves on the unified dataset compared to the simply combined dataset. |
UkraiNER: A New Corpus and Annotation Scheme towards Comprehensive Entity Recognition (2024.lrec-main)
Copied to clipboard
| Challenge: | Named entity recognition excludes nested, discontinuous, non-named entities in practice . despite attempts to broaden their coverage, the most restrictive variant of NER remains the default . |
| Approach: | They propose a new annotation scheme that offers higher comprehensiveness while preserving simplicity. |
| Outcome: | The proposed scheme offers higher comprehensiveness while preserving simplicity . it also includes an annotation tool to implement the scheme on the corpus UkraiNER . |
Unveiling Strengths and Weaknesses of NLP Systems Based on a Rich Evaluation Corpus: The Case of NER in French (2024.lrec-main)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is an applicative task for which annotation schemes vary . a lack of robustness of some tools towards textual variation limits evaluation . |
| Approach: | They propose a gold corpus for french annotated with a rich tagset that enables comparison with multiple annotation schemes. |
| Outcome: | The proposed framework enables a fair comparison of NER systems across textual genres and annotation schemes. |